Noise reduction apparatus, method and program for controlling same, image sensor and digital camera

ABSTRACT

Image data pixels indicative of the pixels in a noise-reduction target area having a size of 5×5 pixels is extracted from a plurality of types of CCD-RAW data having red, green and blue color components. A filter for reducing uncorrelated noise is calculated. Uncorrelated noise is removed by performing a filter operation using the calculated filter while correlativity of the CCD-RAW data is maintained. These processing steps are repeated for one frame of CCD-RAW data. After uncorrelated noise has been removed, spatial pixel processing such as an aperture correction is applied.

This application is a Divisional of co-pending application Ser. No.11/704,344, filed on Feb. 9, 2007, the entire contents of which arehereby incorporated by reference and for which priority is claimed under35 U.S.C. §120.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a noise reduction apparatus, a method andprogram for controlling this apparatus, and an image sensing device anddigital camera having this apparatus.

2. Description of the Related Art

CCDs used in digital still cameras are continuing to be improved interms of number of pixels and sensitivity. The influence of noise,therefore, has become a problem.

Using a low-pass filter or median filter, etc., to remove noise from avideo signal obtained by image sensing has been considered (see thespecification of Japanese Patent Application Laid-Open No. 2004-235472).The removal of noise from an image without detracting from imagesharpness has also been considered (see the specification of JapanesePatent Application Laid-Open No. 2002-222416). There is also prior artfor detecting an edge based upon neighboring pixels and suppressing adecline in resolution (see the specification of Japanese PatentApplication Laid-Open No. 2005-303731).

It is still difficult, however, to remove noise completely.

SUMMARY OF THE INVENTION

Accordingly, an object of the present invention is to reduce noise.

According to a first aspect of the present invention, the foregoingobject is attained by providing a noise reduction apparatus comprising:a noise removal device (means) for removing uncorrelated noise containedin a plurality of types of color image data while maintainingcorrelativity of each of the items of color image data in the pluralityof types of color image data, which represent substantially identicalcolor images of a subject obtained by sensing the image of the samesubject using a plurality of solid-state electronic image sensingdevices; and a pixel processing device (means) for executing spatialpixel processing with regard to the plurality of types of color imagedata from which uncorrelated noise has been removed by the noise removaldevice.

The first aspect of present invention also provides a control methodsuited to the noise reduction apparatus described above. Specifically,the present invention provides a method of controlling a noise reductionapparatus comprising the steps of: removing uncorrelated noise containedin a plurality of types of color image data while maintainingcorrelativity of each of the items of color image data in the pluralityof types of color image data, which represent substantially identicalcolor images of a subject obtained by sensing the image of the samesubject using a plurality of solid-state image sensing devices; andexecuting spatial pixel processing with regard to the plurality of typesof color image data from which uncorrelated noise has been removed.

The first aspect of present invention also provides a program forexecuting the above-mentioned noise reduction processing, as well as animage sensor and digital camera having the noise reduction apparatusdescribed above.

In accordance with the first aspect of the present invention, aplurality of types of color image data representing substantiallyidentical color images of a subject obtained by sensing the image of thesame subject using a plurality of solid-state image sensing devices areobtained. Uncorrelated noise contained in the plurality of types ofcolor image data is removed while maintaining correlativity of each ofthe items of color image data in the plurality of types of color imagedata. Spatial pixel processing is applied to the plurality of types ofcolor image data from which uncorrelated noise has been removed.Uncorrelated noise is removed before spatial pixel processing isexecuted. Therefore, in comparison with a case where uncorrelated noiseis removed after spatial pixel processing is executed, noise reductionprocessing can be achieved without dependence on circuitry such as asignal processing circuit that operates after execution of spatial pixelprocessing. Spatial pixel processing refers to signal processing(interpolation processing, processing for changing pixel level utilizingweighting, digital filter processing, etc.) which, using pixels thatform the image of a subject, is applied to other pixels forming theimage of the subject. Use may be made of one of the pixels, which formone of the images of a subject among a plurality of types of subjectimages represented by a plurality of types of color image data, toexecute signal processing with regard to the other pixels that form thisone image of the subject, or to execute signal processing with regard toother pixels that form another image of a subject different from the oneimage of the subject.

The plurality of types of color image data may have color componentsthat are different from one another.

The plurality of color images of the subject represented by theplurality of types of color image data is such that phasecharacteristics (a characteristic representing spatial position; acharacteristic indicating a deviation in position in a case where aplurality of images of the subject are superimposed) of the pixels thatform respective ones of the plurality of color images of the subject areidentical or exhibit a deviation of less than pixel pitch, by way ofexample.

The apparatus may further comprise a noise removal processingsuppression device for suppressing noise removal processing by the noiseremoval device in accordance with the level of color image data.

It may be so arranged that uncorrelated noise removal processing in thenoise removal device is executed with regard to at least one of acharacteristic of the solid-state electronic image sensing devices andshooting information used when a subject is shot using the solid-stateelectronic image sensing devices.

The plurality of types of color image data may have color componentsthat are different from one another. In this case, the noise removaldevice includes, by way of example, a color image data shifting device(means) for shifting the levels of the plurality of types of color imagedata in such a manner that an average value of the levels of theplurality of types of color image data will become the position of theorigin of color space of the plurality of color components; a filteringdevice (means) for subjecting color image data, which has been shiftedby the color image data shifting device, to processing for removinguncorrelated noise in accordance with the level of this color imagedata; and a color image data reverse shifting device (means) forreturning, in accordance with the amount of shift, the level of thecolor image data from which uncorrelated noise has been removed by thefiltering device.

Noise removal processing in the filtering device is, e.g., digitalfiltering processing that utilizes calculation conforming to the numberof the plurality of color components.

Noise removal processing in the filtering device is, e.g., digitalfiltering processing for handling the plurality of color components insuch a manner that the number of the plurality of color components willbecome less than the original number of color components, with use beingmade of calculation conforming to the number of the plurality of colorcomponents.

The apparatus may further comprise a noise removal processing haltingdevice (means) for halting noise removal processing in a case where theoperation in the digital filter processing has diverged.

The apparatus may further comprise an image data dividing device (means)for dividing (classifying) the plurality of types of color image datainto a plurality of color image data groups including color image dataexhibiting correlativity with one another. Color image group hascorrelativity, since the color image group contains the plurality ofcolor image data exhibiting correlativity with one another. In thiscase, the color image data shifting device would shift a color imagedata group (or color image data groups), which contains color image datathat is to undergo removal of uncorrelated noise, from among theplurality of color image data groups obtained by division in the imagedata dividing device.

The image data dividing device may include a small-block dividing device(means) for dividing (classifying) the plurality of types of color imagedata into a plurality of small blocks. In this case, each of theplurality of small blocks obtained by division in the small-blockdividing device would be divided (classified) into a plurality of colorimage data groups exhibiting correlativity.

The plurality of small blocks may be divided (classified) into aplurality of color image data groups exhibiting correlativity, basedupon the average pixel level of the small blocks obtained by division bythe small-block dividing device.

The plurality of small blocks may be divided (classified) into aplurality of color image data groups exhibiting correlativity, basedupon a pixel level obtained by application of a weighted mean to each ofthe pixels forming the small blocks obtained by division in thesmall-block dividing device, the weighted mean using weightingcoefficients such that the pixel level becomes the average pixel levelas sensitivity of image sensing increases, and becomes arepresentative-pixel level of pixels forming the small blocks assensitivity decreases.

It can also be so arranged that the plurality of small blocks aredivided (classified) into a plurality of color image data groupsexhibiting correlativity, based upon representative pixels thatconstitute the small blocks.

The apparatus may further comprise a noise removal halting device forhalting uncorrelated noise removal processing by the noise removaldevice in a case where the size of an image area represented by thecolor image data group obtained by division in the image data dividingdevice is less than a first predetermined value.

In a case where the size of an image area represented by the color imagedata group is equal to or greater than a second predetermined valuelarger than the first predetermined value, a color image data group isshifted, this group being one which contains color image data that is toundergo removal of uncorrelated noise from among the plurality of colorimage data groups obtained by division in the image data dividingdevice.

Preferably, in a case where the size of an image area represented by thecolor image data group is equal to or greater than a third predeterminedvalue larger than the second predetermined value, uncorrelated noiseremoval processing by the noise removal device is halted and noise ineach item of color image data is removed based upon the differencebetween a dispersion value of each item of color image data containingnoise and a dispersion value of the noise.

The apparatus may further comprise a luminance data generating device(means) for generating luminance data from the color image data. In thiscase, the color image data dividing device would divide (classify) theluminance data, which has been generated by the luminance datagenerating device, into a plurality of luminance data groups, whichexhibit correlativity.

The apparatus may further comprise a block dividing device (means) fordividing (classifying) the color image data of respective ones of theplurality of items of color image data into a plurality ofclose-together blocks in color space; and a level difference calculationdevice (means) for a calculating level difference with respect to acentral pixel among pixels that constitute each of the blocks obtainedby division by the block dividing device. In this case, image data,which represents a collection of pixels for which level differencecalculated by the level difference calculation device is less than apredetermined value, would be shifted.

The level difference calculation device may include: a detection device(means) for detecting the direction in which the slope of a neighboringarea, which contains a target pixel for which the level difference withrespect to the center pixel is calculated, is low; and an average levelcalculation device (means) for calculating an average level of pixels,inclusive of the target pixel, that are present in the direction of thelow slope is detected to be low by the detecting device. In this case, alevel difference between the average level calculated by the averagelevel calculation device and the level of the target pixel is calculatedby the level difference calculation device.

The detecting device may include a high-pass filter for extractinghigh-frequency components in the vertical direction, horizontaldirection, northwest direction and northeast direction of the targetpixel and pixels around the target pixel. In this case, the direction inwhich the slope is low is detected by the detecting device based uponthe high-frequency components extracted by the high-pass filter device.

According to a second aspect of the present invention, the foregoingobject is attained by providing a noise reduction apparatus comprising:a dividing device (means) for dividing (classifying) color image datarepresenting one frame of a color image into a plurality of blocks inwhich pixels are spatially close together; a calculation device (means)for calculating a difference between a dispersion value of block colorimage data representing an image in a block obtained by division in thedividing device and a dispersion value of noise in the block color imagedata; a noise removal device (means) for removing noise in the colorimage data based upon the difference calculated by the calculationdevice; and a control device (means) for exercising control in such amanner that calculation processing by the calculation device and noiseremoval processing in the noise removal device will be repeated withregard to the color image data representing one frame of the colorimage.

The second aspect of present invention also provides a control methodsuited to the noise reduction apparatus described above. Specifically,the present invention provides a method of controlling a noise reductionapparatus comprising the steps of: dividing (classifying) color imagedata representing one frame of a color image into a plurality of blocksin which pixels are spatially close together; calculating a differencebetween a dispersion value of block color image data representing animage in a block obtained by division and a dispersion value of noise inthe block color image data; removing noise in the color image data basedupon the difference calculated; and repeating difference calculationprocessing and noise removal processing with regard to the color imagedata representing one frame of the color image.

The second aspect of present invention also provides a program forimplementing the above-described method of controlling the noisereduction apparatus.

In accordance with the second aspect of the present invention, one frameof a color image is divided into a plurality of blocks in which pixelsare spatially close together. The difference between the dispersionvalue of block color image data (inclusive of noise), which representsthe image in the block, and a dispersion value of noise in the blockcolor image data is calculated. Noise in the color image data is removedbased upon the difference calculated. Such difference calculationprocessing and noise removal processing is repeated for the equivalentof one frame. In a case where noise reduction has been performed takingcolor correlation into account, there are instances where noise cannotbe removed effectively when color correlation happens to occuraccidentally. In accordance with the second aspect of the presentinvention, however, the fact that color correlation regarding colorimage data is not taken into consideration means that it is possible toprevent noise from not being removed owing to the presence of colorcorrelation.

According to a third aspect of the present invention, the foregoingobject is attained by providing a noise reduction apparatus comprising:an extraction device (means) for inputting image data representing oneframe of an image constituted by a number of pixels and extracting imagedata representing a pixel within a zone in which the color image data isregarded as having correlativity; a noise removal device (means) forremoving uncorrelated noise while maintaining correlativity of the imagedata extracted by the extraction device; and a control device (means)for exercising control in such a manner that extraction processing bythe extraction device and noise removal processing by the noise removaldevice is repeated with regard to the image data representing one frameof the image.

The third aspect of present invention also provides a control methodsuited to the noise reduction apparatus described above. Specifically,the present invention provides a method of controlling a noise reductionapparatus comprising the steps of: inputting image data representing oneframe of an image constituted by a number of pixels and extracting imagedata representing a pixel within a zone in which the color image data isregarded as having correlativity; removing uncorrelated noise whilemaintaining correlativity of the image data extracted; and exercisingcontrol in such a manner that extraction processing and noise removalprocessing is repeated with regard to the image data representing oneframe of the image.

The third aspect of present invention also provides a program forimplementing the above-described method of controlling the noisereduction apparatus.

In accordance with the present invention, image data representing animage within a zone in which the color image data is regarded as havingcorrelativity is extracted. Uncorrelated noise is removed whilecorrelativity of the extracted image data is maintained. Sinceuncorrelated noise is removed in a state in which correlativity ismaintained, circuitry that executes image processing with regard toimage data in a state in which the correlativity thereof is maintainedcan be utilized subsequently. The image data may be monochrome imagedata or may be monochromatic color image data obtained from each ofthree solid-state electronic image sensing devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates part of the photoreceptor surface of a CCD;

FIG. 2 illustrates part of the photoreceptor surface of a CCD;

FIG. 3 illustrates part of the photoreceptor surface of a CCD;

FIG. 4 illustrates part of the photoreceptor surface of a CCD;

FIG. 5 illustrates part of the photoreceptor surface of a CCD;

FIG. 6 illustrates part of the photoreceptor surface of a CCD;

FIG. 7 illustrates the manner in which pixels are staggered;

FIG. 8 illustrates the relationship between color space and image data;

FIG. 9 illustrates the relationship between color space and image data;

FIG. 10 illustrates the relationship between color space and image data;

FIG. 11 illustrates the relationship between color space and image data;

FIG. 12 illustrates the structure of an image sensor;

FIG. 13 is a block diagram illustrating the electrical structure of adigital still camera;

FIG. 14 illustrates the relationship between amount of noise and pixellevel;

FIG. 15 is a block diagram illustrating the electrical structure of anoise reduction circuit;

FIG. 16 is a block diagram illustrating the electrical structure of acomputer system;

FIG. 17 is a flowchart illustrating noise reduction processing;

FIG. 18 illustrates an example of the image of a subject;

FIG. 19 illustrates the relationship between color space and image data;

FIG. 20 illustrates an example of an area to undergo noise reduction;

FIG. 21 illustrates an example of an area to undergo noise reduction;

FIG. 22 is a flowchart illustrating processing for area discriminationand the like;

FIG. 23 is a flowchart illustrating part of processing for areadiscrimination and the like;

FIG. 24 illustrates an example of an area to undergo noise reduction;

FIG. 25 illustrates an example of an area to undergo noise reduction;

FIG. 26 illustrates an example of pixels constituting a small block;

FIG. 27 illustrates the relationship between shooting sensitivity andluminance;

FIG. 28 is a flowchart illustrating a process for deciding weightingcoefficients;

FIG. 29 illustrates an example of a pixel array;

FIG. 30 illustrates an example of a pixel array;

FIG. 31 illustrates part of the photoreceptor surface of a CCD;

FIG. 32 illustrates part of the photoreceptor surface of a CCD;

FIG. 33 illustrates part of the photoreceptor surface of a CCD; and

FIG. 34 illustrates pixel division processing.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A preferred embodiment of the present invention will now be described indetail with reference to the drawings.

The principles of noise reduction processing according to a preferredembodiment of the invention will be described first.

In this embodiment, use is made of an image sensor that internallyaccommodates CCDs on three chips (the number of chips need notnecessarily be three, and it will suffice if there are a plurality ofCCD chips).

FIGS. 1 to 3 illustrate parts of the photoreceptor surfaces of CCDs onthree chips constructing the image sensor. FIG. 1 illustrates part ofthe photoreceptor surface of a first CCD that outputs a video signal ofa red color component, FIG. 2 illustrates part of the photoreceptorsurface of a second CCD that outputs a video signal of a green colorcomponent, and FIG. 3 illustrates part of the photoreceptor surface of athird CCD that outputs a video signal of a blue color component.

As shown in FIG. 1, 4096 photodiodes 2R are disposed in the columndirection of the first CCD 1 and 3072 photodiodes 2 in the row directionthereof. Image data representing the image of a subject composed of 4096pixels in the column direction and 3072 pixels in the row direction isobtained by sensing the image of the subject using the first CCD 1. Thevideo signal obtained from each photodiode 2R corresponds individuallyto each pixel of the image.

Formed on the photoreceptor surface of each photodiode 2R of themultiplicity thereof is a color filter having a characteristic thatpasses the red color component. In order to indicate that the filter isone that passes the red color component, the photodiode 2R is assignedthe character “R”.

In this embodiment, noise reduction processing is executed using an area(a noise-reduction target area AR1), which is composed of fivephotodiodes 2R in each of the column and row directions, for a total of25 photodiodes 2R, as a single unit.

When the image of the subject is sensed using the first CCD, the firstCCD outputs CCD-RAW data representing the red-component image of thesubject. The CCD-RAW data appears serially one row's worth of thephotodiodes 2R at a time.

As shown in FIGS. 2 and 3, here also 4096 photodiodes 2G and 2B and 3072of these photodiodes 2G and 2B are disposed in the column and rowdirections, respectively, of the second and third CCDs 2 and 3,respectively. Video signals representing respective ones of subjectimages having 4096 pixels in the column direction and 3072 pixels in therow direction are obtained by sensing the image of the subject using thesecond and third CCDs. The video signals obtained from respective onesof the photodiodes 2G and 2B correspond individually to each pixel ofthe images.

Formed on the photoreceptor surfaces of each of the photodiodes 2G and2B of the multiplicity of photodiodes 2G and 2B are color filters havingcharacteristics that pass the green and blue components, respectively.In order to indicate that the filters are ones that pass the green andblue color components, the photodiodes 2G and 2B are assigned thecharacters “G” and “B”, respectively.

In this embodiment, the areas of 25 photodiodes 2G and 2B each composedof five diodes in each of the column and row directions are illustratedas noise-reduction target areas AG1 and AB1, respectively.

When the image of the subject is sensed using the second and third CCDs,CCD-RAW data representing the green-component image of the subject isobtained from the second CCD, and CCD-RAW data representing theblue-component image of the subject is obtained from the third CCD.

Pixels Rc, Bc and Gc at the centers of the noise-reduction target areasAR1, AB1 and AG1, respectively, are pixels that are to undergo noisereduction. As will be described later in greater detail, noise reductionprocessing of the center pixels Rc, Bc, and Gc is executed utilizing thepixels (image data) present in the noise-reduction target areas AR1, AB1and AG1. When noise reduction processing of the center pixels Re, Bc andGc of the noise-reduction target areas AR1, AB1 and AG1, respectively,ends, the noise-reduction target areas AR1, AB1 and AG1 are each shiftedone pixel to the right and noise reduction processing is applied to thepixels Rc, Bc and G1 c located at the centers of respective ones of thenoise-reduction target areas AR1, AB1 and AG1 thus shifted. Shifting ofthe noise-reduction target areas and noise reduction processing are thusrepeated for one frame of the image.

Since the first to third CCDs shown in FIGS. 1 to 3 are CCDs on threechips constituting the image sensor, the photodiodes 2R, 2G and 2Bconstructing respective ones of the CCDs are considered to be present atthe same position spatially. The pixels represented by the image dataobtained from each of the photodiodes 2R, 2G and 2B represent the samelocation in the images of the subject. In consideration of the fact thatthe pixels representing the subject images of the three types of colorcomponents obtained from the first to third CCDs, namely thered-component pixel (obtained from a photodiode 2R of the first CCD)forming the red-component image of the subject, the green-componentpixel (obtained from a photodiode 2G of the second CCD) forming thegreen-component image of the subject and the blue-component pixel(obtained from a photodiode 2B of the third CCD) forming theblue-component image of the subject, are at substantially the samespatial position, a pixel within the noise-reduction target areas AR1,AB1 and AG1 can be expressed by Xn=(Rn,Gn,Bn) (since there are 25 pixelsin each of the noise-reduction target areas AR1, AB1 and AG1, n=1 to 25holds).

FIGS. 4 to 6 illustrate another example of CCD on three chips. Thisexample is a so-called “honeycomb” array. FIG. 4 illustrates part of thephotoreceptor surface of a first CCD that outputs a video signal of thered color component, FIG. 5 illustrates part of the photoreceptorsurface of a second CCD that outputs a video signal of the green colorcomponent, and FIG. 6 illustrates part of the photoreceptor surface of athird CCD that outputs a video signal of the blue color component.

The first to third CCDs illustrated in FIGS. 4 to 6 are such that inodd-numbered rows, the photodiodes 3R, 3G and 3B are arrayed only in theodd-numbered columns, and in even-numbered rows, the photodiodes 3R, 3Gand 3B are arrayed only in the even-numbered columns. Alternatively,however, in odd-numbered rows, the photodiodes may be arrayed only inthe even-numbered columns, and in even-numbered rows, the photodiodesmay be arrayed only in the odd-numbered columns. The first to third CCDseach comprise 4096 columns and 3072 rows, by way of example.

Provided on the photoreceptor surface of each photodiode 3R of the firstCCD shown in FIG. 4 is a filter “R” having a characteristic that passeslight of the red color component, provided on the photoreceptor surfaceof each photodiode 3G of the second CCD shown in FIG. 5 is a filter “G”having a characteristic that passes light of the green color component,and provided on the photoreceptor surface of each photodiode 3B of thethird CCD shown in FIG. 6 is a filter “B” having a characteristic thatpasses light of the blue color component. By sensing the image of asubject using the first, second and third CCDs illustrated in FIGS. 4, 5and 6, respectively, video signals representing subject images of thered, green and blue color components are obtained.

Areas AR2, AG2, AB2 defined in the column and row directions so as tocontain the 25 photodiodes 3R, 3G or 3B are stipulated asnoise-reduction areas. Pixels Rc, Gc and Bc that are to undergo noisereduction are situated at the approximate centers of the noise-reductionareas AR2, AG2 and AB2, respectively. Noise reduction processing isapplied to one frame's worth of an image while the noise-reduction areasAR2, AG2 and AB2 are moved one pixel at a time in the column and rowdirections in a manner similar to that shown in FIGS. 1 to 3. If thepixels Rc, Gc and Bc that are to undergo noise reduction are situatedsubstantially at the centers of the areas AR2, AG2 and AB2, they neednot be centered perfectly. Of course, these pixels can also be set atthe central positions perfectly depending upon how the areas AR2, AG2and AB2 are defined.

As mentioned above, in consideration of the fact that the pixelsrepresenting the subject images of the three types of color componentsobtained from the first to third CCDs, namely the red-component pixelforming the red-component image of the subject, the green-componentpixel forming the green-component image of the subject and theblue-component pixel forming the blue-component image of the subject,are at substantially the same spatial position, a pixel within thenoise-reduction target areas AR1, AB1 and AG1 can be expressed byXn=(Rn,Gn,Bn) in the CCDs shown in FIGS. 4 to 6 as well.

The CCDs on the three chips shown in FIGS. 4 to 6 need not be positionedso as to reside perfectly at the same spatial position. For example,even if each CCD is shifted by less than the pixel pitch (the spacingbetween the photodiodes), it can be regarded as residing atsubstantially the same spatial position, and noise reduction processingaccording to this embodiment can be executed in the manner describedbelow.

FIG. 7 illustrates the manner in which the spatial positions of thefirst to third CCDs shown in FIGS. 4 to 6 are staggered by a distancethat is half the pixel pitch.

In a case where the first to third CCDs are superimposed virtually, thephotodiodes 3R, 3G and 3B are shifted by a distance that is half thepixel pitch without overlapping one another. Thus, even if the spatialpositions of the photodiodes do not coincide perfectly, noise reductionprocessing described below can be executed so long as they can beconsidered to be substantially coincide.

FIGS. 8 to 11 illustrate relationships between color space of the red,blue and green color components and image data representing a pixel in anoise-reduction target area.

FIG. 8 illustrates the relationship between the color space and imagedata representing a pixel in a noise-reduction target area A0 in a casewhere there is no uncorrelated noise.

The noise-reduction target area A0 is regarded as one havingcorrelativity. A video signal (image data) representing a pixelXn0=(Rn0, G1 no, Bn0, G2 n 0) within the noise-reduction target area A0,therefore, falls within bounds in which the levels of the image data arecomparatively consolidated. Average data of pixel Xn0 within thenoise-reduction target area A0 is indicated by XAV0.

FIG. 9 illustrates the relationship between the color space and imagedata representing a pixel in a noise-reduction target area A1 in a casewhere uncorrelated noise is present. The noise-reduction target area A0for the case where there is no uncorrelated noise also is illustratedfor the purpose of comparison.

In a case where uncorrelated noise is present, image data representing apixel Xn1=(Rn1, G1 n 1, Bn1) within the noise-reduction target area A1is such that the levels of the image data representing each of thepixels are dispersed owing to the uncorrelated noise. Consequently, thezone of the image data representing pixels in the noise-reduction targetarea A1 in a case where uncorrelated noise is present is broader thanthe zone of image data representing pixels in the noise-reduction targetarea A0 in a case where uncorrelated noise is absent. Further, theaverage data of pixel Xn1 in the noise-reduction target area A1 isindicated by XAV1.

Noise reduction processing according to this embodiment eliminatesuncorrelated noise.

FIG. 10 illustrates the relationship between the color space and imagedata representing a pixel in the noise-reduction target area A1 at thetime of movement of image-data coordinates and filtering for removal ofnoise.

In a case where noise reduction is executed in this embodiment, all ofthe image data representing pixel Xn1 in noise-reduction target area A1undergoes a coordinate shift (level shift) in such a manner that thelevel of the average data XAV1 of pixel Xn1 in noise-reduction targetarea A1 will become the origin of the color space. Filtering serving asnoise reduction processing is applied to the image data representing thepixel Xn1 in the noise-reduction target area A1 in a state in which allof the image data representing pixel Xn1 in noise-reduction target areaA1 has undergone a coordinate shift in such a manner that the averagedata XAV1 is shifted to the origin position. Since filtering is appliedwith the average data XAV1 as the position of the origin, comparativelyappropriate filtering can be performed.

FIG. 11 illustrates the relationship between the color space and imagedata representing a pixel in the noise-reduction target area A1 at thetime of reverse movement of image-data coordinates.

The above-described filtering processing eliminates uncorrelated noise.Owing to this filtering processing, the zone of image data representinga noise-reduction target pixel within the noise-reduction target area A1falls within (approaches) the zone of image data representing a pixelwithin the noise-reduction target area A0 in a case where there is nouncorrelated noise. By repeating similar processing also with regard tothe remaining pixels in the noise-reduction target area A1, all of thepixels in the noise-reduction target area A1 fall within (approach) thezone of the image data representing pixels in the noise-reduction targetarea A0.

After the image data representing the pixel Xn1 in the noise-reductiontarget area A1 is subjected to filtering as noise reduction processingin a state in which all of the image data representing the pixel Xn1 inthe noise-reduction target area A1 has had its coordinates shifted, asmentioned above, a coordinate reverse-shift (a level reverse-shift) isperformed in such a manner that the average data XAV1 returns to theposition that prevailed prior to the shift of coordinates. Noisereduction processing is thus completed.

FIG. 12 illustrates an example of an image sensor 30 used in a digitalstill camera according to this embodiment.

The image sensor 30 includes three CCDs 31, 32 and 33. The three CCDs31, 32 and 33 are secured to a separating prism 34 that splits incidentlight into three portions.

When a light beam L representing the image of a subject impinges uponthe image sensor 30, the light beam L is introduced to the separatingprism 34. The light beam L is split in three directions by theseparating prism 34 so as to impinge upon the photoreceptor surfaces ofthe first, second and third CCDs, 31, 32 and 33, respectively.

Provided on the photoreceptor surface of each photodiode of the firstCCD 31 is a filter that passes the red light component, as illustratedin FIG. 1 or FIG. 4. The first CCD 31 outputs an R video signalrepresenting the red color component image of the subject.

Provided on the photoreceptor surface of each photodiode of the secondCCD 32 is a filter that passes the green light component, as illustratedin FIG. 2 or FIG. 5. The second CCD 32 outputs a G video signalrepresenting the green color component image of the subject.

Provided on the photoreceptor surface of each photodiode of the thirdCCD 33 is a filter that passes the blue light component, as illustratedin FIG. 3 or FIG. 6. The third CCD 33 outputs a B video signalrepresenting the blue color component image of the subject.

FIG. 13 is a block diagram illustrating the electrical structure of adigital still camera in which the above-described noise reductionprocessing is executed.

The image sensor 30 of the digital still camera uses the arrangementshown in FIG. 12. By sensing the image of a subject, R, G and B videosignals are output from the image sensor 30, as described above, and areinput to an analog/digital converting circuit 3. The latter outputs theabove-mentioned CCD-RAW data of the red, green and blue colorcomponents.

In a case where CCD-RAW recording has been set by a recording modeswitch (not shown), the CCD-RAW data that has been output from theanalog/digital converting circuit 3 is input to a recording controlcircuit 10. The CCD-RAW data is recorded on a memory card 11 by therecording control circuit 10.

In a case where recording of compressed data has been set by therecording mode switch, the CCD-RAW data that has been output from theanalog/digital converting circuit 3 is input to a color balanceadjustment circuit 4 and is subjected to a color balance adjustment. TheCCD-RAW data that has undergone the color balance adjustment issubjected to a gamma correction in a gamma correction circuit 5 and isthen input to a noise reduction circuit 6.

The noise reduction circuit 6 executes noise reduction based upon theabove-described principle. The noise reduction processing in noisereduction circuit 6 will be described later in greater detail.

The image data that has been output from the noise reduction circuit 6is input to an aperture correction (sharpness correction) circuit (acircuit for executing spatial pixel processing in which the relationshipof pixel positions changes) 7, and a correction such as emphasis ofhigh-frequency components is applied to the image data. Noise reductionprocessing for removing uncorrelated noise in a state in whichcorrelativity is maintained ends in the noise reduction circuit 6 beforethe correction is applied by the aperture correction circuit 7. Byperforming the correction in the aperture correction circuit 7,therefore, uncorrelated noise is removed even if the correlativity ofthe image data is lost.

It will suffice if the noise reduction processing by the noise reductioncircuit 6 is executed before execution of spatial pixel processing suchas the processing executed by the aperture correction circuit 7; thenoise reduction processing may be executed before the adjustment in thecolor balance adjustment circuit 4 or before the correction in the gammacorrection circuit 5, etc. Further, the spatial pixel processing is notlimited solely to an aperture correction and may include processing forperforming resizing that utilizes weighting, processing for reducingnoise further using a low-pass filter, etc. Even in a case where thesetypes of processing are executed, it will suffice if noise reductionprocessing is executed before such processing.

The image data that has been output from the noise reduction circuit 6is input to a YC generating circuit 8. The latter executes processingfor generating luminance data and color difference data. The generatedluminance data and color difference data is compressed in a compressioncircuit 9. The compressed luminance data and color difference data isrecorded on the memory card 11 by the recording control circuit 10.

Further, a luminance data generating circuit 12 may be provided in orderto generate luminance data from the CCD-RAW data that has been outputfrom the gamma correction circuit 5 and apply noise reduction processingusing the luminance data generated. Noise reduction processing utilizingluminance data will be described later in greater detail (see FIGS. 20and 21).

The amount of noise used in noise reduction processing will be describedbefore the details of noise reduction processing.

FIG. 14 illustrates the relationship between amount of noise used innoise reduction processing and pixel values.

In noise reduction processing according to this embodiment, the image ofa prescribed reference subject is sensed and an amount Dn of noise isanalyzed in advance for every pixel level of each of a red colorcomponent, blue color component and green color component.

As the pixel levels of the red color component, blue color component andgreen color component become greater, noise amounts D_(nR), D_(nB) andD_(nG) of these color components, respectively, gradually increase andpeak at certain values. If the pixel values increase further, then thenoise amounts D_(nR), D_(nB) and D_(nG) gradually decrease.

This relationship between the noise amounts D_(nR), D_(nB) and D_(nG)and pixel levels of each of the color components is analyzed beforehandand stored. The stored noise amounts D_(nR), D_(nB) and D_(nG) areutilized in noise reduction processing, described later.

Although the above-mentioned noise amounts D_(nR), D_(nB) and D_(nG) canalso be utilized as is, gain WBG of the color balance adjustment circuit4 may be utilized. A noise amount D_(nG)(□) in a case where the gain WBGof the color balance adjustment circuit 4 is utilized is represented byEquation (1) below.D _(nG)(□)=D _(nG) ×[WBG] ^(□)  Equation (1)

In Equation (1), the gain WBG of color balance adjustment circuit 4 ismultiplied by □ because CCD-RAW data following a □ conversion issubjected to noise reduction processing in this embodiment. It goeswithout saying that in a case where noise reduction processing isapplied to CCD-RAW data prior to the □ conversion, gain WBG notmultiplied by □ is utilized. In a case where noise reduction processingis applied to CCD-RAW data before the color balance adjustment, thenoise amount D_(nG) itself is utilized.

The noise amounts D_(nB)(□) and D_(nR)(□) of the blue and red colorcomponents are represented by Equations (2) and (3) below, in which[WBG] in Equation (1) has been replaced by [WBG(R/G)] and [WBG(B/G)],respectively.D _(nR)(□)=D _(nR) ×[WBG(R/G)]^(□)  Equation (2)D _(nB)(□)=D _(nB) ×[WBG(B/G)]^(□)  Equation (3)

It goes without saying that amounts of noise can be calculated usingimaging information other than color balance. Examples of imaginginformation are the characteristics of the image sensing device such asthe image sensor 30, a shading characteristic, ISO sensitivity, □characteristic, SR combining ratio, dynamic-range characteristic, anautomatic sensitivity increase when a flash of light is emitted in acase where use is made of an electronic flash, shutter speed, EV value,LV value, number of recorded pixels, pixel mode, reproduction band,f-stop number, color difference matrix, lens distortion, zoom position,F-value and contour correction value, etc.

FIG. 15 is a block diagram illustrating the electrical structure of thenoise reduction circuit 6.

When the CCD-RAW data that has been output from the gamma correctioncircuit 5 enters the noise reduction circuit 6, the data is input to afiltering-target extraction circuit 22. The latter extracts CCD-RAWdata, which represents pixels in noise-reduction target areas eachhaving five pixels in both the column and row directions (these areasare represented as the noise-reduction target areas AR1, AB1 and AG1 inFIGS. 1 to 3), from the CCD-RAW data of every color component.

The CCD-RAW data to undergo filtering is input to a filter calculationcircuit 23. The latter calculates a filter F in accordance with Equation(4) below.F={D _((s+n)) −αD _(n) }D _((s+n)) ⁻¹   Equation (4)where D_((s+n)) in Equation (4) is a quantity that contains a signal andnoise and is represented by Equation (5) below. Further, □ is a filtercontrol coefficient.

$\begin{matrix}{D_{({s + n})} = \begin{bmatrix}D_{{({s + n})}R} & D_{{({s + n})}{RG}} & D_{{({s + n})}{RB}} \\D_{{({s + n})}{GR}} & D_{{({s + n})}G} & D_{{({s + n})}{GB}} \\D_{{({s + n})}{BR}} & D_{{({s + n})}{BG}} & D_{{({s + n})}B}\end{bmatrix}} & {{Equation}\mspace{14mu}(5)}\end{matrix}$

Here a diagonal component of D_((s+n)) is amount of variance of thesignal of each color, and a non-diagonal component is amount of varianceof a signal between colors. D_((s+n)x)(x=R, B, G) is represented byEquation (6) below, and the non-diagonal component D_((s+n)x1gx2)(x1,x2=R, B, G) is represented by Equation (7) below.

Further, Dn in Equation (4) is a quantity solely of noise and isrepresented by Equation (6) below.

$\begin{matrix}{D_{n} = \begin{bmatrix}D_{nR} & 0 & 0 \\0 & D_{nG} & 0 \\0 & 0 & D_{nB}\end{bmatrix}} & {{Equation}\mspace{14mu}(6)}\end{matrix}$

When the filter F is thus calculated, the calculated filter and theCCD-RAW data representing the pixels in the noise-reduction target areaare input to a filter operation circuit 24. The latter performs a filteroperation (noise reduction processing) that is based on Equation (7)below.

$\begin{matrix}{\begin{bmatrix}R_{out} \\G_{out} \\B_{out}\end{bmatrix} = {{F\begin{bmatrix}{R_{c} - {avR}} \\{G_{c} - {avG}} \\{B_{c} - {avB}}\end{bmatrix}} + \begin{bmatrix}{avR} \\{avG} \\{avB}\end{bmatrix}}} & {{Equation}\mspace{14mu}(7)}\end{matrix}$

In Equation (7), Rout, Gout and Bout indicate image data of the redcolor component, green color component and blue color component,respectively, obtained following the filter operation; Rc, Gc and Bcindicate image data representing noise-reduction target pixels presentat the centers of the noise-reduction target areas AR1, AG1 and AB1 ofthe red color component, green color component and blue color component,respectively, obtained by color-component division processing; and avR,avG and avB are items of data indicating average values of image data ofpixels in the noise-reduction target areas AR1, AG1 and AB1 of the redcolor component, green color component and blue color component,respectively, obtained by color-component division processing.

In Equation (7), the levels of image data of pixels in thenoise-reduction target areas AR, AG1 and AB1 are shifted in the mannerdescribed above (see FIG. 10) to the origin position of the color spacethat has the red color component, green color component and blue colorcomponent as its coordinate system by subtracting the data avR, avG andavB indicating the average values of image data of pixels in thenoise-reduction target areas AR1, AG1 and AB1 of the red colorcomponent, green color component and blue color component, respectively,from the noise-reduction target pixels Rc, Gc and Bc present at thecenters of the noise-reduction target areas AR1, AG1 and AB1 of the redcolor component, green color component and blue color component thathave been obtained by color-component division processing. The thusshifted pixels Rc, Gc and Bc to undergo noise reduction are subjected tofiltering processing using the filter F indicated by Equation (4).

When D_(n) indicated by Equation (6) is subtracted from D_((S+n))indicated by Equation (5) in the filtering processing using the filter Findicated by Equation (4), the uncorrelated noise of the CCD-RAW data iseliminated because the diagonal component of D_((s+n)) is subtracted,and the correlativity of the CCD-RAW data is maintained because thenon-diagonal component is not subtracted. In other words, uncorrelatednoise is removed while the correlativity of the CCD-RAW data ismaintained.

By adding the data avR, avG and avB indicating the average values to thenoise-reduction target pixels Rc, Gc and Bc obtained following filteringprocessing, the noise-reduction target pixels Rc, Gc and Bc are returnedto levels corresponding to the original levels from which noise has beenreduced.

In order to prevent parameters from becoming too large and, hence,prevent degradation of the image following noise reduction processing inthe above-described noise reduction processing, it is preferred thatparameters D(s+n)x and Dnx used in Equations (1) to (5) satisfy therelation of Equation (8) below. Here x=R, G, B holds.if D _((s+n)x) <D _(nx), then D _((s+n)x) =D _(nx)   Equation (8)

Such noise reduction processing is repeated with regard to one frame'sworth of image data. Of course, if the level of color image data (thelevel of a pixel to undergo noise reduction) is equal to or greater thana predetermined level, noise reduction processing may be halted.

The image data that has undergone filter processing in the filteroperation circuit 24 is input to an array reverse-conversion processingcircuit 25. The array of image data that has been divided on aper-color-component basis is returned from the array of the colorfilters of CCD 1 to the original array of color filters of CCD 1 in thearray reverse-conversion processing circuit 25. The output of the arrayreverse-conversion processing circuit 25 is the output of the noisereduction circuit 6.

Although a covariance matrix is utilized in the noise reductionprocessing described above, noise reduction processing can also beexecuted as described next without utilizing a covariance matrix.

A filter F is represented by Equation (9) in a manner similar to that ofEquation (4).

$\begin{matrix}{F = {\frac{D_{{({s + n})}X} - D_{nx}}{D_{{({s + n})}X}} = \frac{\sigma_{{({s + n})}X}^{2} - {\alpha \cdot \sigma_{nX}^{2}}}{\sigma_{{({s + n})}X}^{2}}}} & {{Equation}\mspace{14mu}(9)}\end{matrix}$where X=R, G, B holds.

In Equation (9), D(s+n)x is a statistical quantity (a variance value,the square of a standard deviation) that contains a signal and noise; itis represented by Equation (10).D _((s+n)X)=□² _((s+n)X)   Equation (10)where X=R, G, B holds.

Further, Dnx in Equation (9) is a statistical quantity that containsnoise only and is represented by Equation (11).D _(nX)=□² _(nX)   Equation (11)where X=R, G, B holds.

If the filter F based upon Equation (9) is calculated, the calculatedfilter and CCD-RAW data representing pixels within the noise-reductiontarget area are input to the filter operation circuit 24. The latterexecutes a filter operation (noise reduction processing) that is basedupon Equation (12).X _(out) =F·(X _(in) −avX)+avX   Equation (12)where X=R, G, B holds.

In Equation (12), Xout indicates respective ones of image data of thered, green and blue color components that result after the filteroperation. Further, Xin indicates respective ones of image datarepresenting noise-reduction target pixels of the red, green and bluecolor components. In addition, avR, avG and avB are items of dataindicating average values of image data of pixels in the noise-reductiontarget areas AR1, AG1 and AB1 of the red, green and blue colorcomponents, respectively, as described earlier.

Whereas color correlation is taken into consideration in the noisereduction processing executed utilizing Equations (4) to (7), it is nottaken into consideration in the noise reduction processing of Equations(9) to (12). In a case where noise reduction processing has beenexecuted taking color correlation into consideration, there areinstances where the effectiveness of noise reduction cannot be expectedthat much in portions of an image where color correlation happens tooccur accidentally. However, in the noise reduction processing thatutilizes Equations (9) to (12), noise reduction is improved even inportions of an image in which color correlation occurs accidentally.This noise reduction processing is effective especially in flat imageportions and monochromatic image portions. Further, since theseequations are comparatively simple, the circuitry for implementing themcan be reduced in scale.

FIG. 16 is a block diagram illustrating the electrical structure of acomputer system.

The above-described embodiment is such that noise reduction processingis executed in a digital still camera. However, noise reductionprocessing of CCD-RAW data can also be executed utilizing a computersystem.

The computer system includes a CPU 40 to which have been connected adisplay unit 41, a printer 42, a keyboard 43 and a memory 44.

Also connected to the CPU 40 is a memory card reader/writer 45. Byloading a memory card 51 on which CCD-RAW data has been recorded intothe memory card reader/writer 45, the CCD-RAW data is read from thememory card 51 and the data is subjected to the above-described noisereduction processing. A CD-ROM drive 46 is further connected to the CPU40. If a CD-ROM 52 on which a program for the above-described noisereduction processing has been stored is loaded into the CD-ROM drive 46,the noise reduction processing program will be read from the CD-ROM 52and installed in the computer. Noise reduction processing can be appliedto the CCD-RAW data that has been read from the memory card 51.

Further connected to the CPU 40 is a hard-disk drive 47. CCD-RAW datathat has undergone noise reduction processing can be recorded on a harddisk 48 by the hard-disk drive 47.

FIG. 17 is a flowchart illustrating noise reduction processing executedin the computer system.

As mentioned above, CCD-RAW data having red, green and blue colorcomponents is obtained from the memory card 51. Image data to undergonoise reduction is extracted from this CCD-RAW data. (step 32).

The filter F is calculated (step 33) and processing is executed usingthe filter F calculated, whereby uncorrelated noise is removed (step34). The processing from step 32 to step 34 is repeated until the filteroperation has been applied to the final pixel of one frame of the image(step 35). This is followed by executing the above-described processingfor reverse conversion of the array (step 36). Spatial pixel processingsuch as the above-mentioned aperture correction is executed afteruncorrelated noise is thus removed.

In the embodiment described above, a filter operation in noise reductionprocessing is executed using color components of the three colors.However, a filter operation can also be executed using 2 colors×2 colorcomponents. For example, using the same green color components,components are divided into a set of red and green color components anda set of blue and green color components. Calculation can be performedin the manner set forth next with regard to the set of red and greencolor components. It goes without saying that calculation can beperformed in the same manner with regard to the set of blue and greencolor components.

In the case where 2 colors×2 color components are used, Equations (5)and (6) cited above are represented by Equations (13) and (14) below,respectively.

$\begin{matrix}{D_{({s + n})} = \begin{bmatrix}D_{{({s + n})}R} & D_{{({s + n})}{RG}} \\D_{{({s + n})}{GR}} & D_{{({s + n})}G}\end{bmatrix}} & {{Equation}\mspace{14mu}(13)} \\{D_{n} = \begin{bmatrix}D_{nR} & 0 \\0 & D_{nG}\end{bmatrix}} & {{Equation}\mspace{14mu}(14)}\end{matrix}$

Equation (7) cited above becomes Equation (15) when Equations (9) and(14) are used.

$\begin{matrix}{\begin{bmatrix}R_{out} \\G_{out}\end{bmatrix} = {{F\begin{bmatrix}{R_{c} - {avR}} \\{G_{c} - {avG}}\end{bmatrix}} + \begin{bmatrix}{avR} \\{avG}\end{bmatrix}}} & {{Equation}\mspace{14mu}(15)}\end{matrix}$

Accordingly, in a case where an inverse matrix at the time ofcalculation of F is not found, it is preferred to so arrange it thatnoise reduction processing is not executed.

For example, in a case where a filter operation based upon Equation (4)is performed using Equation (13), D_((s+n)) ⁻¹ is represented byEquation (16) below.

$\begin{matrix}{D_{({s + n})}^{- 1} = {\frac{1}{{AD} - {BC}}\begin{bmatrix}D & {- B} \\{- C} & A\end{bmatrix}}} & {{Equation}\mspace{14mu}(16)}\end{matrix}$where Equation (17) below is written for D_((s+n)) indicated by Equation(6).

$\begin{matrix}{D_{({s + n})} = {\begin{bmatrix}D_{{({s + n})}R} & D_{{({s + n})}{RG}} \\D_{{({s + n})}{GR}} & D_{{({s + n})}G}\end{bmatrix} = \begin{bmatrix}A & B \\C & D\end{bmatrix}}} & {{Equation}\mspace{14mu}(17)}\end{matrix}$

Assume that Δ=AD−BC holds in Equation (16). If the value of Δ approacheszero, the image represented by image data that has undergone noisereduction processing will be degraded and therefore noise reductionprocessing is inhibited.

FIGS. 18 to 22 illustrate a modification of the embodiment.

FIG. 18 illustrates an example of the image of a subject represented byCCD-RAW data.

Depending upon the camera angle, there are occasions where an edge 63 isproduced in the image 60 of the subject. There are instances wherecorrelativity vanishes between the level of the image data representingthe image in an image area 61 on the left side of the edge 63 and thelevel of the image data representing the image in an image area 62 onthe right side.

FIG. 19 illustrates a level distribution of image data representingpixels within a noise-reduction target area in the color space of red,blue and green color components.

If the edge 63 is produced in the image 60 of the subject and there isno image correlativity between the areas 61 and 62 on both sides of theedge 63, as shown in FIG. 12, the distribution of image data of pixelsin a noise-reduction target area AE that includes the edge 63 becomesscattered in accordance with the position of a pixel on the image 60 ofthe subject in the noise-reduction target area without falling withinfixed bounds centered on the average value XAV of image datarepresenting pixels in the noise-reduction target area, as mentionedabove. Consequently, even if the average value XAV is shifted to theorigin in color space and filtering processing is executed in the mannerdescribed above, there are instances where the correlativity of pixelsin the noise-reduction target area AE cannot be maintained.

FIG. 20 illustrates an example of a noise-reduction target area AY.

In this modification, CCD-RAW data is converted to luminance data andthe luminance data obtained by the conversion is used.

The noise-reduction target area AY contains a total of 25 pixels 72,namely five pixels in each of the column and row directions. A pixel Ycat the center of these pixels is a pixel to undergo noise reductionprocessing.

In a case where the noise-reduction target area AY involves the edge 63in the manner described above, there are instances where noise reductionprocessing cannot be executed while correlativity between pixels ismaintained. In this modification, therefore, as shown in FIG. 21, anoise-reduction target area AY1 exhibiting correlativity is detectedanew and the above-described noise reduction processing is executedusing the average value of pixels in the new noise-reduction target areaAY detected. Noise reduction processing can be executed whilecorrelativity of pixels is maintained.

FIG. 22 is a flowchart illustrating processing for detecting anoise-reduction target area having correlativity.

Luminance data within a noise-reduction target area having five pixelsin each of column and row directions is generated from CCD-RAW data(step 81; see FIG. 20). It is determined whether the difference betweenthe level of image data of one pixel (a pixel to undergo discrimination)within the noise-reduction target area and the level of image data ofthe pixel Yc at the center is less than a predeterminedarea-discrimination threshold value (step 82). If the difference is lessthan the threshold value (“YES” at step 82), then it is construed thatthis one pixel has correlativity with respect to the center pixel Yc.Accordingly, this one pixel, which is the pixel undergoingdiscrimination, is added to a new noise-reduction target area (step 83).If the difference is equal to or greater than the threshold value (“NO”at step 82), then it is construed that this one pixel does not havecorrelativity with respect to the center pixel. This pixel, therefore,is not added to the new noise-reduction target area. The above-describedprocessing of steps 82 and 83 is repeated with regard to all pixels toundergo discrimination (step 84). If the pixel processed is not thefinal pixel (“NO” at step 84), then a pixel neighboring (e.g., on theright side) the one pixel is set as a new pixel to undergodiscrimination (step 85).

If the above-described processing of steps 82 and 83 has been completedfor all pixels (“YES” at step 84), then the new noise-reduction targetarea AY1 is decided as illustrated in FIG. 21.

The method of processing changes further in accordance with the size ofthe new noise-reduction target area AY1 thus decided.

In a case where the size of the new noise-reduction target area AY1 isvery small and is less than a first threshold value (“YES” at step 86),the number of pixels within the noise-reduction target area will be toosmall and there will be instances where comparatively appropriate noisereduction processing cannot be executed. As a result, noise reductionprocessing is halted (step 87).

If the size of the noise-reduction target area AY1 is greater than thefirst threshold value (“NO” at step 86) but is comparatively small andtherefore is smaller than a second threshold value (first thresholdvalue □ second threshold value) (“YES” at step 88), then, in order toenlarge the number of pixels in the noise-reduction target area, noisereduction processing is not executed using the new noise-reductiontarget area AY1 but is executed using a noise-reduction target area of aprescribed size (step 89).

If the size of the new noise-reduction target area AY1 is greater thanthe second threshold value (“NO” at step 88), then it is construed thatthe noise-reduction target area of the prescribed size contains an edgeportion or the like. In order to eliminate the edge portion, noisereduction processing is executed using the new noise-reduction targetarea AY1 (step 90).

The above-described processing for deciding a new noise-reduction targetarea performs discrimination one pixel at a time. However, it may be soarranged that the decision is rendered for every block of a plurality ofpixels.

FIG. 23 is a flowchart illustrating a modification of processing shownin FIG. 22.

Processing in FIG. 23 identical with that shown in FIG. 22 is identifiedby like step numbers and need not be described again.

The processing illustrated in FIG. 22 is such that if the size of thearea decided as the noise-reduction target area is larger than thesecond threshold value (“NO” at step 88), then noise reductionprocessing is applied to the new noise reduction area decided. In themodification illustrated in FIG. 23, however, it is decided whether thesize of the area that has been decided as the noise reduction area issmaller than (larger than) a third threshold value that is larger thanthe second threshold value (step 91).

If the new noise reduction area that has been decided is larger than thethird threshold value (“NO” at step 91), then this area is considered tobe a portion having a comparatively large flatness or a portion in whichchange is smooth. Noise reduction processing that is based uponEquations (9) to (12) above (linear noise reduction processing) isexecuted in the manner described above (step 92).

If the new noise reduction area decided is smaller than the thirdthreshold value (“YES” at step 91), then this area is considered to be aportion containing high frequency or a portion comparatively close to anedge. Noise reduction processing that is based upon Equations (4) to (7)(noise reduction processing using a covariance matrix) is executed (step93) in the manner described above.

FIG. 24 illustrates an example of a noise-reduction target area.

As shown in FIG. 24, pixels in a noise-reduction target area are dividedinto individual blocks of every row and column with the exception of therow and column that contain the center pixel Yc. That is, in the rowdirection, the pixels are divided into small blocks P2, P6, P8 and P4 ofa first row, second row, fourth row (the third row is excluded becauseit contains the center pixel Yc) and fifth row, respectively. In thecolumn direction, the pixels are divided into small blocks P3, P7, P5and P1 of a first column, second column, fourth column (the third columnis excluded because it contains the center pixel Yc) and fifth column,respectively.

Among the pixels constituting each small block, the center pixel isadopted as the representative pixel of this small block. Therepresentative pixels of the small blocks P2, P6, P8 and P4 of thefirst, second, fourth and fifth rows, respectively, are Y2, Y6, Y8 andY4, respectively. The representative pixels of the small blocks P3, P7,P5 and P1 of the first, second, fourth and fifth columns, respectively,are Y3, Y7, Y5 and Y1, respectively.

It is determined whether the difference between the level of therepresentative pixel of each block and the level of the center pixel Ycis less than the area-discrimination threshold value, as mentionedabove. If the difference is less than the threshold value, then theentirety of the small block that contains this representative pixel isadded to the new noise-reduction target area.

FIG. 25 illustrates an example of a noise-reduction target area decidedanew.

Pixels of the first row and first column have been excluded from anoise-reduction target area AY2 decided anew. The reason for this isthat with regard to the representative pixels Y2 and Y3, the leveldifference with respect to the center pixel Yc has been determined to beequal to or greater than the predetermined area-discrimination thresholdvalue. With regard to the representative pixels Y6, Y8, Y4, Y7, Y5 andY1 of the small blocks of the other rows and columns, the leveldifference relative to the center pixel is less than the predeterminedarea-discrimination threshold value and therefore the pixels containedin each of these small blocks are contained in the noise-reductiontarget area AY2.

Thus, processing is simplified by discriminating a noise-reductiontarget area on a per-small-block basis.

In the embodiment set forth above, it is determined whether thedifference between the level of the representative pixel of each smallblock and the level of the pixel Yc at the center is less than anarea-discrimination threshold value. However, whether the differencebetween an average level of pixels constituting a small block and thelevel of the pixel Yc at the center, rather than the difference betweenthe level of the representative pixel of each small block and the levelof the pixel Yc at the center, is less than the area-discriminationthreshold value may be discriminated.

FIG. 26 illustrates an example of five pixels constituting a smallblock.

As mentioned above, the small block includes five pixels Xi, Xj, Xk, Xl,Xm. Although these pixels Xi, Xj, Xk, Xl, Xm have been arrayed in asingle horizontal row, they may also be arrayed in a single verticalrow, as described earlier.

If we let the levels of these pixels Xi, Xj, Xk, Xl, Xm be representedby Xi, Xj, Xk, Xl, Xm, respectively, then the average level B1 of thepixels Xi, Xj, Xk, Xl, Xm constituting the small block is represented byEquation (18), where a0 to a4 are weighting coefficients.B1=a0·Xi+a1·Xj+a2·Xk+a3·X1+a4·Xm   Eq. (18)

Thus, it may be so arranged that discrimination processing as to whetherto adopt an area as a noise-reduction target area is executed based uponthe difference between the average level of a small block and the centerpixel Yc. Since the average level of pixels constituting a small blockis used, the amount of noise in pixels also is averaged. In a case wherethe difference between the level of representative pixels of smallblocks and the level of the center pixel Yc has been calculated, thenoise at the level of the representative pixels stands out. Even if thedifference between the level of the representative pixel and the centerpixel Yc is not calculated accurately, a comparatively accuratedifference value can be calculated by using the average level. Anoise-reduction target area can be defined comparatively accurately.

FIGS. 27 and 28 illustrate a modification.

In Equation (18) above, there is no particular discussion concerning theweighting coefficients a0 to a4. In the modification described below,the weighting coefficients a0 to a4 are decided in such a manner thatthe higher the sensitivity of imaging, the more the average level B1 inEquation (18) is averaged.

FIG. 27 illustrates the relationship between level Y of luminance dataand standard deviation □ of noise possessed by the luminance data.

Luminance data Y (image data) obtained by low-sensitivity imaging suchas ISO (International Standards Organization) 100 has a comparativelysmall amount of noise and a noise standard deviation □ that is alsosmall. By contrast, luminance data Y obtained by high-sensitivityimaging such as ISO 1600 has a comparatively large amount of noise and anoise standard deviation □ that is also large.

Since the amount of noise is large in case of high-sensitivity imaging,there are instances where a noise reduction area cannot be decidedaccurately owing to the effects of noise if a pixel for which thedifference in level with respect to the level of the center pixel Yc iscalculated is a representative pixel in a small block. By contrast,since the amount of noise is comparatively small in case oflow-sensitivity imaging, the effects of noise are not that great even ifa pixel for which the difference in level with respect to the level ofthe center pixel Yc is calculated is a representative pixel in a smallblock. Accordingly, the weighting coefficients are decided in such amanner that the level for calculating the difference with respect to thelevel of the center pixel Yc is the average level of pixels constitutingthe small block, with the level approaching the level of arepresentative pixel of the small block (e.g., the pixel at the centerof the small block) as the sensitivity of imaging declines.

FIG. 28 is a flowchart illustrating processing for deciding weightingcoefficients.

In a case where ISO sensitivity of an exposure is 100 (“YES” at step101) and also in a case where it is 200 (“YES” at step 103), it can beconstrued that there is no noise in both cases. Accordingly, only theweighting coefficient a2 regarding a representative pixel of a smallblock is made 1 and the other coefficients a0, a1, a3, a4 are made 0(steps 102, 104) in such a manner that the difference between the levelof the representative pixel of the small block and the level of thepixel Yc at the center will be calculated.

In a case where ISO sensitivity of an exposure is 400 (“YES” at step105), noise increases slightly, the weighting coefficient a2 regardingthe representative pixel (center pixel) of the small block is madecomparatively large (a2=0.5), the weighting coefficients a1, a3 adjacentboth sides of the representative pixel are made comparatively small (a1,a3=0.25), and the other pixel weighting coefficients a0, a4 are made 0(step 106).

In a case where ISO sensitivity of an exposure is 800 (“YES” at step107), noise increases further. Accordingly, the weighting coefficient a2regarding the representative pixel (center pixel) of the small block ismade 0.4, the weighting coefficients a1, a3 adjacent both sides of therepresentative pixel are made 0.2, and the other pixel weightingcoefficients a0, a4 are made 0.1 (step 108).

In a case where ISO sensitivity of an exposure is 1600 (“YES” at step109), noise is great. Accordingly, the weighting coefficients a0 to a4are all made equal, i.e., 0.2, in such a manner that the differencebetween the average level and the level of the pixel Yc at the centerwill be calculated (step 110).

Thus, a noise-reduction target area can be decided comparativelyaccurately in case of both low-sensitivity imaging and high-sensitivityimaging.

FIGS. 29 and 30, which illustrate a modification, show pixel arrays.

In a case where a noise-reduction target area is decided one pixel at atime, the difference in level with respect to the center pixel Yc iscalculated. However, in a case where a target pixel for which the leveldifference with respect to the center pixel Yc is calculated contains alarge amount of noise, there are instances where the level differenceincreases or, conversely, decreases owing to the effects of noise.Consequently, there are occasions where an accurate level differencecannot be calculated. This embodiment calculates the direction in whichthe slope of a target pixel, for which the level difference with respectto the center pixel Yc is calculated, is small (namely the direction inwhich a change in the image is small, or the direction in which theimage is flat and smooth).

FIG. 29 illustrates pixels around a target pixel.

Pixels b0 to b8 in three rows and three columns are illustrated. Acenter pixel b0 is a target pixel. Pixels are arranged around a centralpixel b0, namely in a vertical direction, horizontal direction,northwest direction and northeast direction. High-frequency componentsare extracted by a high-pass filter in each of these directions usingEquations (19) to (22). Equation (19) is utilized in a case wherehigh-frequency components in the horizontal direction are extracted,Equation (20) is utilized in a case where high-frequency components inthe vertical direction are extracted, Equation (21) is utilized in acase where high-frequency components in the northwest direction areextracted, and Equation (22) is utilized in a case where high-frequencycomponents in the northeast direction are extracted.HPF _(H) =|hpf ₀ ·b ₄ +hpf ₁ ·b ₀ +hpf ₂ ·b ₅|  Eq. (19)HPF _(V) =|hpf ₀ ·b ₂ +hpf ₁ ·b ₀ +hpf ₂ ·b ₇|  Eq. (20)HPF _(NW) =|hpf ₀ ·b ₁ +hpf ₁ ·b ₀ +hpf ₂ ·b ₈|  Eq. (21)HPF _(NE) =|hpf ₀ ·b ₃ +hpf ₁ ·b ₀ +hpf ₂ ·b ₆|  Eq. (22)

The direction in which the absolute values of the levels ofhigh-frequency components thus extracted are smallest is the directionin which the slope of the target pixel is small. A mean level LPF ofthree pixels along this direction is calculated according to Equation(23).LPF=(bx+by+bz)/3   Eq. (23)

The difference between the mean level LPF and the level of the centerpixel Yc, and the noise-reduction target area is decided based upon thedifference value. Thus the difference between the mean level LPF and thelevel of the center pixel Yc is calculated. Therefore, even if thetarget pixel contains a comparatively significant amount of noise, thenoise is averaged and the influence that the noise has upon thedifference value can be suppressed. In particular, since the directionin which the slope of the pixel is small has been detected, theexistence of edges and the like can be neglected in comparison with acase where use is made of the average value of pixel levels in thedirection in which the slope is large. This makes it possible to bettersuppress the effects of noise (e.g., if three pixels are averaged, thestandard deviation □ is 1/3√). The threshold value for edgediscrimination can also be lowered, and even edges not found in theprior art can be found.

FIG. 30 illustrates the manner in which it is determined whether thetarget pixel Y1 is included in the noise-reduction target area.

As mentioned above, the direction in which the slope of the target pixelY1 is low is detected using pixels Y00, Y10, Y01, Y2, Y02, Y6, Y7 (thesepixels are indicated by As in FIG. 30) in the vertical, horizontal,northwest and northeast directions of the target pixel Y1. The averageof the three pixels arrayed along the detected direction in which theslope is small is calculated. The difference between the calculatedaverage level and the level of the center pixel Yc is then calculated.If the difference calculated is less than a threshold value, then thetarget pixel Y1 is contained in the noise-reduction target area. Whetherthe pixels Y2 to Y24 other than the target pixel Y1 fall in thenoise-reduction target area is determined in a similar manner.

In the foregoing embodiment, the average of three pixels arrayed alongthe direction in which the slope of the target pixel Y1 is small iscalculated and then the difference between the calculated average leveland the level of the center pixel Yc per se is calculated. If thedifference calculated in less than the threshold value, then it isjudged that the target pixel Y1 is contained in the noise-reductiontarget area. However, it may be so arranged that with regard to thecenter pixel Yc as well, the pixel Yc is adopted as the center in amanner similar to the target pixel Y1, the direction in which the slopeis small is detected using the pixel Yc and the pixels Y7 to Y9, Y12,Y13 and Y16 to Y18 around the pixel Yc, and the average of three pixelsarrayed along the detected direction in which the slope is small (thethree pixels are Y8, Yc and Y17 in a case where the slope in thevertical direction is small) is calculated. It may be so arranged thatif the difference between the calculated average level of the pixel Ycand the average level of the target pixel Y1 is calculated and thecalculated difference is less than a threshold value, then the targetpixel Y1 is judged to fall within the noise-reduction target area.

It goes without saying that the embodiment of the present invention isnot limited to the above-described filter array and can be applied toany filter array. For example, as shown in FIG. 31, the embodiment isalso applicable to a so-called honeycomb array in which a (4n+1)thcolumn, (4n+2)th column, (4n+3)th column and (4n+4)th column areprovided in odd-numbered rows with filters having characteristics thatpass a red color component, first green color component, blue colorcomponent and second green color component, respectively, and the(4n+1)th column, (4n+2)th column, (4n+3)th column and (4n+4)th columnare provided in even-numbered rows with filters having characteristicsthat pass a blue color component, second green color component, redcolor component and first green color component, respectively. Further,as shown in FIG. 32, the embodiment is also applicable to a so-calledBayer array in which odd-numbered rows and odd-numbered columns areprovided with filters having a characteristic that passes the red colorcomponent, odd-numbered rows and even-numbered columns are provided withfilters having a characteristic that passes the first green colorcomponent, even-numbered rows and odd-numbered columns are provided withfilters having a characteristic that passes the second green colorcomponent, and even-numbered rows and even-numbered columns are providedwith filters having a characteristic that passes the blue colorcomponent. Thus, this embodiment is applicable if the array of colorfilters is systematic.

FIG. 33 illustrates part of the photoreceptor surface of a CCD. This CCDis for monochrome imaging.

Arrayed on the surface of the CCD are a multiplicity of photodiodes P11to P14. In odd-numbered columns, the photodiodes P11 to P14 are providedin even-numbered rows, and in even-numbered columns, the photodiodes P11to P14 are provided in odd-numbered rows. However, it goes withoutsaying that it may be arranged such that in odd-numbered columns, thephotodiodes are provided in odd-numbered rows, and in even-numberedcolumns, the photodiodes are provided in even-numbered rows, or suchthat photodiodes are provided in all rows and columns.

In this embodiment, four photodiodes P11 to P14 close together in thecolumn and row directions are handled as one set 120 (it goes withoutsaying that the number need not necessarily be four). Noise reductionprocessing is executed with a substantially square zone (noise-reductiontarget area A) containing 25 of the sets 120 serving as a unit. Althoughthe physical positions (spatial positions) of the four photodiodes P11to P14 that construct each set 120 are different, in these physicalpositions are regarded as being the same in this embodiment.

If the image of a subject is sensed using such a CCD, the CCD willoutput CCD-RAW data representing the image of the subject. The CCD-RAWdata is output serially one row's worth at a time in accordance with thearray of photodiodes P11 to P14.

FIG. 34 illustrates how an image looks after image division.

In this embodiment, data extraction processing (pixel divisionprocessing) of CCD-RAW data is executed in such a manner that an arrayof pixels (this pixel array corresponds to the array of photodiodes P1to P4 in FIG. 33) of an image represented by the CCD-RAW data that hasbeen output from the CCD in the manner described above will represent animage represented by signal charge that has been stored in each of thephotodiodes of the photodiodes P11 to P14 that construct the single set120.

The image represented by signal charge that has been stored in each ofthe photodiodes P11, P12, P13 and P14 is composed of image portions ofan upper-left area IP1, lower-left area IP2, lower-right area IP3 andupper-right area IP4 of FIG. 34.

The areas of five pixels in each of the column and row directions inrespective ones of these image portions IP1 to IP4 becomenoise-reduction target areas AP1 to AP4. The area obtained by combiningthese noise-reduction target areas AP1 to AP4 corresponds to thenoise-reduction target area 120 shown in FIG. 33.

Pixels AP1 c, AP2 c, AP3 c and AP4 c at the centers of thenoise-reduction target areas AP1 to AP4, respectively, are pixels thatare to undergo noise reduction. Noise reduction processing of thecentral pixels AP1 c, AP2 c, AP3 c and AP4 c is executed utilizing thepixels (image data) present in the noise-reduction target areas AP1 toAP4 in a manner similar to that described above. When noise reductionprocessing of the central pixels AP1 c, AP2 c, AP3 c and AP4 c of thenoise-reduction target areas AP1 to AP4, respectively, ends, thenoise-reduction target areas AP1 to AP4 are each shifted one pixel tothe right and noise reduction processing is applied to the pixels AP1 c,AP2 c, AP3 c and AP4 c located at the centers of respective ones of thenoise-reduction target areas AP1 to AP4 thus shifted. Shifting of thenoise-reduction target areas and noise reduction processing are thusrepeated for one frame of the image.

As mentioned above, the positions of four pixels adjoining one anotherin the column and row directions in the CCD 1 differ physically but thepixels are regarded as being at the same position in this embodiment.This means that a pixel within the noise-reduction target areas AP1 toAP4 can be expressed by Xn=(P11 n, P12 n, P13 n, P14 n). (Since thepixels within the noise-reduction target areas AP1 to AP4 are 25 innumber, n=1 to 25 holds.)

It will be understood that by adopting the foregoing stipulations, noisereduction processing can be executed in a manner similar to thatdescribed above with reference to FIG. 9 onward. Equations (5) to (7),however, become Equations (24) to (26) below.

$\begin{matrix}{D_{({s + n})} = \begin{bmatrix}D_{{({s + n})}P\; 11} & D_{{({s + n})}P\; 11P\; 12} & D_{{({s + n})}P\; 11P\; 13} & D_{{({s + n})}P\; 11P\; 14} \\D_{{({s + n})}P\; 12P\; 11} & D_{{({s + n})}P\; 12} & D_{{({s + n})}P\; 12P\; 13} & D_{{({s + n})}P\; 12P\; 14} \\D_{{({s + n})}P\; 13P\; 11} & D_{{({s + n})}P\; 13P\; 12} & D_{{({s + n})}P\; 13} & D_{{({s + n})}P\; 13P\; 14} \\D_{{({s + n})}P\; 14P\; 11} & D_{{({s + n})}P\; 14P\; 12} & D_{{({s + n})}P\; 14P\; 13} & D_{{({s + n})}P\; 14}\end{bmatrix}} & {{Equation}\mspace{14mu}(24)} \\{D_{n} = \begin{bmatrix}D_{{nP}\; 11} & 0 & 0 & 0 \\0 & D_{{nP}\; 12} & 0 & 0 \\0 & 0 & D_{{nP}\; 13} & 0 \\0 & 0 & 0 & D_{{nP}\; 14}\end{bmatrix}} & {{Equation}\mspace{14mu}(25)} \\{\begin{bmatrix}{P\; 11_{out}} \\{P\; 12_{out}} \\{P\; 13_{out}} \\{P\; 14_{out}}\end{bmatrix} = {{F\begin{bmatrix}{{P\; 11_{c}} - {{avP}\; 11}} \\{{P\; 12_{c}} - {{avP}\; 12}} \\{{P\; 13_{c}} - {{avP}\; 13}} \\{{P\; 14_{c}} - {{avP}\; 14}}\end{bmatrix}} + \begin{bmatrix}{{avP}\; 11} \\{{avP}\; 12} \\{{avP}\; 13} \\{{avP}\; 14}\end{bmatrix}}} & {{Equation}\mspace{14mu}(26)}\end{matrix}$

In the foregoing embodiment, the invention is described with regard to amonochrome CCD. However, noise reduction processing can be executed insimilar fashion also in each of CCDs on three chips.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

1. A noise reduction apparatus comprising: a dividing device fordividing color image data representing one frame of a color image into aplurality of blocks in which pixels are spatially close together; acalculation device for calculating a difference between a dispersionvalue of block color image data representing an image in a blockobtained by division by said dividing device and a dispersion value ofnoise in the block color image data; a noise removal device for removingnoise in the color image data based upon the difference calculated bysaid calculation device; and a control device for exercising control insuch a manner that calculation processing by said calculation device andnoise removal processing by said noise removal device will be repeatedwith regard to the color image data representing one frame of the colorimage.
 2. A method of controlling a noise reduction apparatus,comprising the steps of: dividing color image data representing oneframe of a color image into a plurality of blocks in which pixels arespatially close together; calculating a difference between a dispersionvalue of block color image data representing an image in a blockobtained by division and a dispersion value of noise in the block colorimage data; removing noise in the color image data based upon thedifference calculated; and repeating difference calculation processingand noise removal processing with regard to the color image datarepresenting one frame of the color image.
 3. A non-transitory computerreadable medium having stored there on a program that causes a processorof a noise removal apparatus to execute the steps of: dividing colorimage data representing one frame of a color image into a plurality ofblocks in which pixels are spatially close together; calculating adifference between a dispersion value of block color image datarepresenting an image in a block obtained by division and a dispersionvalue of noise in the block color image data; removing noise in thecolor image data based upon the difference calculated; and repeatingdifference calculation processing and noise removal processing withregard to the color image data representing one frame of the colorimage.